Methods and Systems for Detecting Weather Conditions Including Fog Using Vehicle Onboard Sensors

ABSTRACT

Methods and systems for detecting weather conditions including fog using vehicle onboard sensors are provided. An example method includes receiving laser data collected from scans of an environment of a vehicle, and associating, by a computing device, laser data points of with one or more objects in the environment. The method also includes comparing laser data points that are unassociated with the one or more objects in the environment with stored laser data points representative of a pattern due to fog, and based on the comparison, identifying by the computing device an indication that a weather condition of the environment of the vehicle includes fog.

CROSS-REFERENCE TO RELATED APPLICATION

The present disclosure is a continuation of and claims priority to U.S.patent application Ser. No. 13/873,442, filed on Apr. 30, 2013, theentire contents of which are herein incorporated by reference.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Autonomous vehicles use various computing systems to aid in thetransport of passengers from one location to another. Some autonomousvehicles may require an initial input or continuous input from anoperator, such as a pilot, driver, or passenger. Other autonomoussystems, for example autopilot systems, may be used when the system hasbeen engaged, which permits the operator to switch from a manual mode(where the operator exercises a high degree of control over the movementof the vehicle) to an autonomous mode (where the vehicle essentiallydrives itself) to modes that lie somewhere in between.

Such vehicles are typically equipped with various types of sensors inorder to detect objects in the surroundings. For example, an autonomousvehicle may include lasers, sonar, radar, cameras, and other deviceswhich scan and record data from surroundings of the vehicle. Sensor datafrom one or more of these devices may be used to detect objects andtheir respective characteristics (position, shape, heading, speed,etc.). This detection and identification is useful for the safeoperation of autonomous vehicle.

SUMMARY

Within examples, devices and methods for detecting weather conditionsincluding fog using vehicle onboard sensors are provided.

In one example, a method is provided that comprises receiving laser datacollected from scans of an environment of a vehicle, and the laser dataincludes a plurality of laser data points. The method also includesassociating, by a computing device, laser data points of the pluralityof laser data points with one or more objects in the environment, andcomparing laser data points that are unassociated with the one or moreobjects in the environment with stored laser data points representativeof a pattern due to fog. The method further includes based on thecomparison, identifying by the computing device an indication that aweather condition of the environment of the vehicle includes fog.

In another example, a non-transitory computer readable storage mediumhaving stored therein instructions, that when executed by a computingdevice, cause the computing device to perform functions. The functionscomprise receiving laser data collected from scans of an environment ofa vehicle, and the laser data includes a plurality of laser data points.The functions also comprise associating laser data points of theplurality of laser data points with one or more objects in theenvironment, and comparing laser data points that are unassociated withthe one or more objects in the environment with stored laser data pointsrepresentative of a pattern due to fog. The functions further comprisebased on the comparison, identifying by the computing device anindication that a weather condition of the environment of the vehicleincludes fog.

In still another example, a system is provided that comprises at leastone processor, and data storage comprising instructions executable bythe at least one processor to cause the system to perform functions. Thefunctions comprise receiving laser data collected from scans of anenvironment of a vehicle, and the laser data includes a plurality oflaser data points. The functions further comprise associating laser datapoints of the plurality of laser data points with one or more objects inthe environment, and comparing laser data points that are unassociatedwith the one or more objects in the environment with stored laser datapoints representative of a pattern due to fog. The functions furthercomprise based on the comparison, identifying by the computing device anindication that a weather condition of the environment of the vehicleincludes fog.

In still another example, a device is provided comprising a means forreceiving laser data collected for an environment of a vehicle, and thelaser data includes a plurality of laser data points. The device alsocomprises a means for determining laser data points of the plurality oflaser data points that are associated with the one or more objects inthe environment. The device also comprises based on laser data pointsbeing unassociated with the one or more objects in the environment, ameans for identifying by the computing device an indication that aweather condition of the environment of the vehicle includes fog.

These as well as other aspects, advantages, and alternatives, willbecome apparent to those of ordinary skill in the art by reading thefollowing detailed description, with reference where appropriate to theaccompanying figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a functional block diagram depicting a vehicle according to anexample embodiment.

FIG. 2 depicts an example vehicle that can include all or some of thefunctions described in connection with the vehicle in reference to FIG.1.

FIG. 3 is a block diagram of an example method for detecting weatherconditions including fog using onboard vehicle sensors, in accordancewith at least some embodiments described herein.

FIG. 4 is a block diagram of example methods for determining furtherindications of weather conditions including a fog using onboard vehiclesensors, in accordance with at least some embodiments described herein.

FIG. 5A is an example conceptual illustration of identifying anindication that a weather condition of the environment includes fog.

FIG. 5B is an example conceptual illustration of an image captured bythe vehicle in FIG. 5A.

FIGS. 6A-6B include example conceptual side view illustrations ofidentifying an indication that an environment of a vehicle includes afog.

DETAILED DESCRIPTION

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols identify similarcomponents, unless context dictates otherwise, and the figures orcomponents of the figures may not necessarily be drawn to scale forillustration purposes. The illustrative system and method embodimentsdescribed herein are not meant to be limiting. It may be readilyunderstood that certain aspects of the disclosed systems and methods canbe arranged and combined in a wide variety of different configurations,all of which are contemplated herein.

Within examples, methods and systems are provided for detecting weatherconditions using vehicle onboard sensors, and modifying behavior of thevehicle accordingly. In some examples, self-driving cars or autonomousvehicles may not drive or drive as well under certain weather conditionssuch as heavy rain, wet-road, fog, direct sun light, etc., and thus,behavior the autonomous vehicle may be based on the detected weathercondition.

In one example, a method is provided that comprises receiving laser datacollected for an environment of a vehicle, and a computing devicedetermining laser data points of the plurality of laser data points thatare associated with one or more objects in the environment. Based onlaser data points being unassociated with the one or more objects in theenvironment, the computing device can identify an indication that aweather condition of the environment of the vehicle includes fog.

In a specific example, a laser sensor may be unable to detect objectsthrough fog, and may in fact, receive data points reflected off the fog.For any untracked received laser data (e.g., laser data that does notmatch to tracked or known objects in the environment), an indication canbe determined that the vehicle is in an environment that has a foggyweather condition.

Further information may be used to provide a higher confidence level orconfirmation of the weather condition is foggy. For example, informationmay include information about the weather from a server, image data froma camera coupled to the vehicle, or data from a radar sensor that maysee through the fog and detect the objects not seen by the laser.

The indication that the weather condition is foggy can be useful todetermine safe driving actions of an autonomous vehicle. Example actionsmay include providing instructions to indicate a request to transitionto a manual mode, or if remaining in autonomous mode then switching to amode specific to driving in fogs (i.e., driving at slower speeds,allowing for larger distances to accomplish braking, turn on fog lights,etc.).

Example systems within the scope of the present disclosure will now bedescribed in greater detail. Generally, an example system may beimplemented in or may take the form of an automobile. However, anexample system may also be implemented in or take the form of othervehicles, such as cars, trucks, motorcycles, buses, boats, airplanes,helicopters, lawn mowers, recreational vehicles, amusement parkvehicles, farm equipment, construction equipment, trams, golf carts,trains, and trolleys. Other vehicles are possible as well.

FIG. 1 is a functional block diagram depicting a vehicle 100 accordingto an example embodiment. The vehicle 100 is configured to operate fullyor partially in an autonomous mode, and thus may be referred to as an“autonomous vehicle.” For example, a computer system 112 may control thevehicle 100 while in an autonomous mode via control instructions to acontrol system 106 for the vehicle 100. The computer system 112 mayreceive information from a sensor system 104, and base one or morecontrol processes (such as the setting a heading so as to avoid adetected obstacle) upon the received information in an automatedfashion.

The vehicle 100 may be fully autonomous or partially autonomous. In apartially autonomous vehicle some functions can optionally be manuallycontrolled (e.g., by a driver) some or all of the time. Further, apartially autonomous vehicle may be configured to switch between afully-manual operation mode and a partially-autonomous and/or afully-autonomous operation mode.

The vehicle 100 may include various subsystems such as a propulsionsystem 102, a sensor system 104, a control system 106, one or moreperipherals 108, as well as a power supply 110, a computer system 112,and a user interface 116. The vehicle 100 may include more or fewersubsystems and each subsystem may include multiple elements. Further,each of the subsystems and elements of vehicle 100 may beinterconnected. Thus, one or more of the described functions of thevehicle 100 may be divided up into additional functional or physicalcomponents, or combined into fewer functional or physical components. Insome further examples, additional functional and/or physical componentsmay be added to the examples illustrated by FIG. 1.

The propulsion system 102 may include components operable to providepowered motion to the vehicle 100. Depending upon the embodiment, thepropulsion system 102 may include an engine/motor 118, an energy source119, a transmission 120, and wheels/tires 121. The engine/motor 118could be any combination of an internal combustion engine, an electricmotor, steam engine, Stirling engine, or other types of engines and/ormotors. In some embodiments, the propulsion system 102 may includemultiple types of engines and/or motors. For instance, a gas-electrichybrid vehicle may include a gasoline engine and an electric motor.Other examples are possible as well.

The energy source 119 may represent a source of energy that may, in fullor in part, power the engine/motor 118. That is, the engine/motor 118may be configured to convert the energy source 119 into mechanicalenergy to operate the transmission 120. Examples of energy sources 119may include gasoline, diesel, other petroleum-based fuels, propane,other compressed gas-based fuels, ethanol, solar panels, batteries,capacitors, flywheels, regenerative braking systems, and/or othersources of electrical power, etc. The energy source 119 may also provideenergy for other systems of the automobile 100.

The transmission 120 may include elements that are operable to transmitmechanical power from the engine/motor 118 to the wheels/tires 121. Suchelements may include a gearbox, a clutch, a differential, a drive shaft,and/or axle(s), etc. The transmission 120 may include other elements aswell. The drive shafts may include one or more axles that may be coupledto the one or more wheels/tires 121.

The wheels/tires 121 may be arranged to stably support the vehicle 100while providing frictional traction with a surface, such as a road, uponwhich the vehicle 100 moves. Accordingly, the wheels/tires 121 ofvehicle 100 may be configured in various formats, including a unicycle,bicycle/motorcycle, tricycle, or car/truck four-wheel format. Otherwheel/tire geometries are possible, such as those including six or morewheels. Any combination of the wheels/tires 121 of vehicle 100 may beoperable to rotate differentially with respect to other wheels/tires121. The wheels/tires 121 may represent at least one wheel that isfixedly attached to the transmission 120 and at least one tire coupledto a rim of the wheel that could make contact with the driving surface.The wheels/tires 121 may include any combination of metal and rubber, oranother combination of materials.

The sensor system 104 generally includes one or more sensors configuredto detect information about the environment surrounding the vehicle 100.For example, the sensor system 104 may include a Global PositioningSystem (GPS) 122, a precipitation sensor 123, an inertial measurementunit (IMU) 124, a RADAR unit 126 (radio detection and ranging), a laserrangefinder/LIDAR unit 128 (laser imaging detection and ranging), acamera 130, and/or a microphone 131. The sensor system 104 may alsoinclude sensors configured to monitor internal systems of the vehicle100 (e.g., O₂ monitor, fuel gauge, engine oil temperature, wheel speedsensors, etc.). One or more of the sensors included in the sensor system104 may be configured to be actuated separately and/or collectively inorder to modify a position and/or an orientation of the one or moresensors.

Sensors in the sensor system 104 may be configured to provide data thatis processed by the computer system 112 in real-time. For example,sensors may continuously update outputs to reflect an environment beingsensed at or over a range of time, and continuously or as-demandedprovide that updated output to the computer system 112 so that thecomputer system 112 can determine whether the vehicle's then currentdirection or speed should be modified in response to the sensedenvironment.

The GPS 122 may be any sensor configured to estimate a geographiclocation of the vehicle 100. To this end, GPS 122 may include atransceiver operable to provide information regarding the position ofthe vehicle 100 with respect to the Earth.

The precipitation sensor 123 may be mounted under or incorporated into awindshield of the vehicle 100. Precipitation sensors may also be mountedat various other locations, such as at or near a location of headlamps,etc. In one example, the precipitation sensor 123 may include a set ofone or more infrared light-emitting diodes (LEDs) and a photodetectorsuch as a photodiode. Light emitted by the LEDs may be reflected by thewindshield back to the photodiode. The less light the photodiodereceives may be indicative of the more precipitation outside of thevehicle 100. An amount of reflected light or some other indicator of thedetected amount of precipitation may be passed to computer system 112.

The IMU 124 may include any combination of sensors (e.g., accelerometersand gyroscopes) configured to sense position and orientation changes ofthe vehicle 100 based on inertial acceleration.

The RADAR unit 126 may represent a system that utilizes radio signals tosense objects within the local environment of the vehicle 100. In someembodiments, in addition to sensing the objects, the RADAR unit 126 mayadditionally be configured to sense the speed and/or heading of theobjects.

Similarly, the laser rangefinder or LIDAR unit 128 may be any sensorconfigured to sense objects in the environment in which the vehicle 100is located using lasers. Depending upon the embodiment, the laserrangefinder/LIDAR unit 128 could include one or more laser sources, alaser scanner, and one or more detectors, among other system components.The laser rangefinder/LIDAR unit 128 could be configured to operate in acoherent (e.g., using heterodyne detection) or an incoherent detectionmode. The LIDAR unit 128 may have a range of about 150 meters, about athirty degree vertical field of view, and about a 360 degree horizontalfield of view; however, other configurations are possible as well. TheLIDAR unit 128 may provide information indicative of range and intensityinformation useful to identify a location and distance of variousobjects in the vehicle's surroundings. In one aspect, the LIDAR unit 128may provide data to measure a distance between the vehicle 100 andsurfaces of objects facing the vehicle 100 by spinning on its axis andchanging its pitch.

The camera 130 may include one or more devices configured to capture aplurality of images of the environment surrounding the vehicle 100. Thecamera 130 may be a still camera or a video camera. In some embodiments,the camera 130 may be mechanically movable such as by rotating and/ortilting a platform to which the camera is mounted. As such, a controlprocess of the vehicle 100 may be implemented to control the movement ofthe camera 130.

The sensor system 104 may also include a microphone 131. The microphone131 may be configured to capture sound from the environment surroundingthe vehicle 100. In some cases, multiple microphones can be arranged asa microphone array, or possibly as multiple microphone arrays.

The control system 106 may be configured to control operation(s) of thevehicle 100 and its components. Accordingly, the control system 106 mayinclude various elements include steering unit 132, throttle 134, brakeunit 136, a sensor fusion algorithm 138, a computer vision system 140, anavigation/pathing system 142, and an obstacle avoidance system 144,etc.

The steering unit 132 may represent any combination of mechanisms thatmay be operable to adjust the heading of vehicle 100. For example, thesteering unit 132 can adjust the axis (or axes) of one or more of thewheels/tires 121 so as to effect turning of the vehicle 100. Thethrottle 134 may be configured to control, for instance, the operatingspeed of the engine/motor 118 and, in turn, control the speed of thevehicle 100. The brake unit 136 may include any combination ofmechanisms configured to decelerate the vehicle 100. The brake unit 136may, for example, use friction to slow the wheels/tires 121. In otherembodiments, the brake unit 136 inductively decelerates the wheels/tires121 by a regenerative braking process to convert kinetic energy of thewheels/tires 121 to electric current. The brake unit 136 may take otherforms as well.

The sensor fusion algorithm 138 may be an algorithm (or a computerprogram product storing an algorithm) configured to accept data from thesensor system 104 as an input. The data may include, for example, datarepresenting information sensed at the sensors of the sensor system 104.The sensor fusion algorithm 138 may include or be configured to beexecuted using, for instance, a Kalman filter, Bayesian network, orother algorithm. The sensor fusion algorithm 138 may provide variousassessments based on the data from sensor system 104. Depending upon theembodiment, the assessments may include evaluations of individualobjects and/or features in the environment of vehicle 100, evaluationsof particular situations, and/or evaluations of possible impacts basedon the particular situation. Other assessments are possible.

The computer vision system 140 may be any system operable to process andanalyze images captured by camera 130 in order to identify objectsand/or features in the environment of vehicle 100 that could includetraffic signals, road way boundaries, other vehicles, pedestrians,and/or obstacles, etc. The computer vision system 140 may use an objectrecognition algorithm, a Structure From Motion (SFM) algorithm, videotracking, and other computer vision techniques. In some embodiments, thecomputer vision system 140 could be additionally configured to map anenvironment, track objects, estimate the speed of objects, etc.

The navigation and pathing system 142 may be any system configured todetermine a driving path for the vehicle 100. For example, thenavigation/pathing system 142 may determine a series of speeds anddirectional headings to effect movement of the vehicle 100 along a paththat substantially avoids perceived obstacles while generally advancingthe vehicle 100 along a roadway-based path leading to an ultimatedestination, which may be set according to user inputs via the userinterface 116, for example. The navigation and pathing system 142 mayadditionally be configured to update the driving path dynamically whilethe vehicle 100 is in operation. In some embodiments, the navigation andpathing system 142 could be configured to incorporate data from thesensor fusion algorithm 138, the GPS 122, and one or more predeterminedmaps so as to determine the driving path for vehicle 100.

The obstacle avoidance system 144 may represent a control systemconfigured to identify, evaluate, and avoid or otherwise negotiatepotential obstacles in the environment of the vehicle 100. For example,the obstacle avoidance system 144 may effect changes in the navigationof the vehicle 100 by operating one or more subsystems in the controlsystem 106 to undertake swerving maneuvers, turning maneuvers, brakingmaneuvers, etc. In some embodiments, the obstacle avoidance system 144is configured to automatically determine feasible (“available”) obstacleavoidance maneuvers on the basis of surrounding traffic patterns, roadconditions, etc. For example, the obstacle avoidance system 144 may beconfigured such that a swerving maneuver is not undertaken when othersensor systems detect vehicles, construction barriers, other obstacles,etc. in the region adjacent the vehicle 100 that would be swerved into.In some embodiments, the obstacle avoidance system 144 may automaticallyselect the maneuver that is both available and maximizes safety ofoccupants of the vehicle. For example, the obstacle avoidance system 144may select an avoidance maneuver predicted to cause the least amount ofacceleration in a passenger cabin of the vehicle 100.

The control system 106 may additionally or alternatively includecomponents other than those shown and described.

The vehicle 100 also includes peripherals 108 configured to allowinteraction between the vehicle 100 and external sensors, othervehicles, other computer systems, and/or a user, such as an occupant ofthe vehicle 100. For example, the peripherals 108 for receivinginformation from occupants, external systems, etc. may include awireless communication system 146, a touchscreen 148, a microphone 150,and/or a speaker 152.

In some embodiments, the peripherals 108 function to receive inputs fora user of the vehicle 100 to interact with the user interface 116. Tothis end, the touchscreen 148 can both provide information to a user ofthe vehicle 100, and convey information from the user indicated via thetouchscreen 148 to the user interface 116. The touchscreen 148 can beconfigured to sense both touch positions and touch gestures from thefinger of a user (or stylus, etc.) via capacitive sensing, resistancesensing, optical sensing, a surface acoustic wave process, etc. Thetouchscreen 148 can be capable of sensing finger movement in a directionparallel or planar to the touchscreen surface, in a direction normal tothe touchscreen surface, or both, and may also be capable of sensing alevel of pressure applied to the touchscreen surface. An occupant of thevehicle 100 can also utilize a voice command interface. For example, themicrophone 150 can be configured to receive audio (e.g., a voice commandor other audio input) from an occupant of the vehicle 100. Similarly,the speaker 152 can be configured to output audio to the occupant of thevehicle 100.

In some embodiments, the peripherals 108 function to allow communicationbetween the vehicle 100 and external systems, such as devices, sensors,other vehicles, etc. within its surrounding environment and/orcontrollers, servers, etc., physically located far from the vehicle 100that provide useful information regarding the vehicle's surroundings,such as traffic information, weather information, etc. For example, thewireless communication system 146 can wirelessly communicate with one ormore devices directly or via a communication network. The wirelesscommunication system 146 can optionally use 3G cellular communication,such as CDMA, EVDO, GSM/GPRS, and/or 4G cellular communication, such asWiMAX or LTE. Additionally or alternatively, the wireless communicationsystem 146 can communicate with a wireless local area network (WLAN),for example, using WiFi. In some embodiments, the wireless communicationsystem 146 could communicate directly with a device, for example, usingan infrared link, short-range wireless link, etc. The wirelesscommunication system 146 can include one or more dedicated short rangecommunication (DSRC) devices that can include public and/or private datacommunications between vehicles and/or roadside stations. Other wirelessprotocols for sending and receiving information embedded in signals,such as various vehicular communication systems, can also be employed bythe wireless communication system 146 within the context of the presentdisclosure.

The power supply 110 may provide power to components of the vehicle 100,such as electronics in the peripherals 108, the computer system 112, thesensor system 104, etc. The power supply 110 can include a rechargeablelithium-ion or lead-acid battery for storing and discharging electricalenergy to the various powered components, for example. In someembodiments, one or more banks of batteries may be configured to provideelectrical power. In some embodiments, the power supply 110 and theenergy source 119 can be implemented together, as in some all-electriccars.

Many or all of the functions of the vehicle 100 may be controlled viathe computer system 112 that receives inputs from the sensor system 104,the peripherals 108, etc., and communicates appropriate control signalsto the propulsion system 102, the control system 106, the peripherals108, etc. to effect automatic operation of the vehicle 100 based on itssurroundings. The computer system 112 may include at least one processor113 (which could include at least one microprocessor) that executesinstructions 115 stored in a non-transitory computer readable medium,such as the data storage 114. The computer system 112 may also representa plurality of computing devices that may serve to control individualcomponents or subsystems of the vehicle 100 in a distributed fashion.

In some embodiments, data storage 114 may contain instructions 115(e.g., program logic) executable by the processor 113 to execute variousautomobile functions, including those described above in connection withFIG. 1. Data storage 114 may contain additional instructions as well,including instructions to transmit data to, receive data from, interactwith, and/or control one or more of the propulsion system 102, thesensor system 104, the control system 106, and the peripherals 108.

In addition to the instructions 115, the data storage 114 may store datasuch as roadway maps, path information, among other information. Suchinformation may be used by vehicle 100 and computer system 112 at duringthe operation of the vehicle 100 in the autonomous, semi-autonomous,and/or manual modes.

The vehicle 100, and associated computer system 112, providesinformation to and/or receives input from, a user of the vehicle 100,such as an occupant in a passenger cabin of the vehicle 100.Accordingly, the vehicle 100 may include a user interface 116 forproviding information to or receiving input from a user of vehicle 100.The user interface 116 may control or enable control of content and/orthe layout of interactive images that could be displayed on thetouchscreen 148. Further, the user interface 116 could include one ormore input/output devices within the set of peripherals 108, such as thewireless communication system 146, the touchscreen 148, the microphone150, and the speaker 152.

The computer system 112 controls the operation of the vehicle 100 basedon inputs received from various subsystems indicating vehicle and/orenvironmental conditions (e.g., propulsion system 102, sensor system104, and/or control system 106), as well as inputs from the userinterface 116, indicating user preferences. For example, the computersystem 112 may utilize input from the control system 106 to control thesteering unit 132 to avoid an obstacle detected by the sensor system 104and the obstacle avoidance system 144. The computer system 112 may beconfigured to control many aspects of the vehicle 100 and itssubsystems. Generally, however, provisions are made for manuallyoverriding automated controller-driven operation, such as in the eventof an emergency, or merely in response to a user-activated override,etc.

The components of the vehicle 100 described herein may be configured towork in an interconnected fashion with other components within oroutside their respective systems. For example, the camera 130 cancapture a plurality of images that represent information about anenvironment of the vehicle 100 while operating in an autonomous mode.The environment may include other vehicles, traffic lights, trafficsigns, road markers, pedestrians, etc. The computer vision system 140can categorize and/or recognize various aspects in the environment inconcert with the sensor fusion algorithm 138, the computer system 112,etc. based on object recognition models pre-stored in the data storage114, and/or by other techniques.

Although FIG. 1 shows various components of vehicle 100, i.e., wirelesscommunication system 146, computer system 112, data storage 114, anduser interface 116, as being integrated into the vehicle 100, one ormore of these components could be mounted or associated separately fromthe vehicle 100. For example, data storage 114 could, in part or infull, exist separate from the vehicle 100. Thus, the vehicle 100 couldbe provided in the form of device elements that may be locatedseparately or together. The device elements that make up vehicle 100 maygenerally be communicatively coupled together in a wired and/or wirelessfashion.

FIG. 2 depicts an example vehicle 200 that can include all or some ofthe functions described in connection with the vehicle 100 in referenceto FIG. 1. Although example vehicle 200 is illustrated in FIG. 2 as afour-wheel sedan-type car for illustrative purposes, the presentdisclosure is not so limited. For instance, example vehicle 200 canrepresent any type of vehicle.

Example vehicle 200 includes a sensor unit 202, a wireless communicationsystem 204, a LIDAR unit 206, a laser rangefinder unit 208, and a camera210. Furthermore, example vehicle 200 may include any of the componentsdescribed in connection with vehicle 100 of FIG. 1.

The sensor unit 202 is mounted atop example vehicle 200 and includes oneor more sensors configured to detect information about an environmentsurrounding example vehicle 200, and output indications of theinformation. For example, the sensor unit 202 may include anycombination of cameras, RADARs, LIDARs, range finders, and acousticsensors. The sensor unit 202 may include one or more movable mounts thatmay be operable to adjust the orientation of one or more sensors in thesensor unit 202. In one embodiment, the movable mount may include arotating platform that may scan sensors so as to obtain information fromeach direction around example vehicle 200. In another embodiment, themovable mount of the sensor unit 202 may be moveable in a scanningfashion within a particular range of angles and/or azimuths. The sensorunit 202 may be mounted atop the roof of a car, for instance, howeverother mounting locations are possible. Additionally, the sensors of thesensor unit 202 may be distributed in different locations and need notbe collocated in a single location. Some possible sensor types andmounting locations include the LIDAR unit 206 and laser rangefinder unit208. Furthermore, each sensor of the sensor unit 202 may be configuredto be moved or scanned independently of other sensors of the sensor unit202.

The wireless communication system 204 may be located on a roof ofexample vehicle 200 as depicted in FIG. 2. Alternatively, the wirelesscommunication system 204 may be located, fully or in part, elsewhere.The wireless communication system 204 may include wireless transmittersand receivers that may be configured to communicate with devicesexternal or internal to example vehicle 200. Specifically, the wirelesscommunication system 204 may include transceivers configured tocommunicate with other vehicles and/or computing devices, for instance,in a vehicular communication system or a roadway station. Examples ofsuch vehicular communication systems include dedicated short rangecommunications (DSRC), radio frequency identification (RFID), and otherproposed communication standards directed towards intelligent transportsystems.

The camera 210 may be a photo-sensitive instrument, such as a stillcamera, a video camera, etc., that is configured to capture a pluralityof images of the environment of example vehicle 200. To this end, thecamera 210 can be configured to detect visible light, and canadditionally or alternatively be configured to detect light from otherportions of the spectrum, such as infrared or ultraviolet light. Thecamera 210 can be a two-dimensional detector, and can optionally have athree-dimensional spatial range of sensitivity. In some embodiments, thecamera 210 can include, for example, a range detector configured togenerate a two-dimensional image indicating distance from the camera 210to a number of points in the environment. To this end, the camera 210may use one or more range detecting techniques.

For example, the camera 210 may provide range information by using astructured light technique in which example vehicle 200 illuminates anobject in the environment with a predetermined light pattern, such as agrid or checkerboard pattern and uses the camera 210 to detect areflection of the predetermined light pattern from environmentalsurroundings. Based on distortions in the reflected light pattern,example vehicle 200 may determine the distance to the points on theobject. The predetermined light pattern may comprise infrared light, orradiation at other suitable wavelengths for such measurements.

The camera 210 may be mounted inside a front windshield of examplevehicle 200. Specifically, the camera 210 may be situated to captureimages from a forward-looking view with respect to the orientation ofexample vehicle 200. Other mounting locations and viewing angles of thecamera 210 may also be used, either inside or outside example vehicle200.

The camera 210 can have associated optics operable to provide anadjustable field of view. Further, the camera 210 may be mounted toexample vehicle 200 with a movable mount to vary a pointing angle of thecamera 210, such as a via a pan/tilt mechanism.

FIG. 3 is a block diagram of an example method for detecting weatherconditions including fog using onboard vehicle sensors, in accordancewith at least some embodiments described herein. Method 300 shown inFIG. 3 presents an embodiment of a method that, for example, could beused with the vehicle 100 and/or vehicle 200 as illustrated anddescribed in reference to FIGS. 1 and 2, respectively, or components ofthe vehicle 100 or vehicle 200. For example, the processes describedherein may be carried out by the RADAR unit 126, the LIDAR unit 128, orcamera 130 mounted to an autonomous vehicle (e.g., vehicle 200) incommunication with the computer system 112, the sensor fusion algorithm138, and/or the computer vision system 140. Method 300 may include oneor more operations, functions, or actions as illustrated by one or moreof blocks 302-306. Although the blocks are illustrated in a sequentialorder, these blocks may in some instances be performed in parallel,and/or in a different order than those described herein. Also, thevarious blocks may be combined into fewer blocks, divided intoadditional blocks, and/or removed based upon the desired implementation.

In addition, for the method 300 and other processes and methodsdisclosed herein, the flowchart shows functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions (e.g., machine readable code)executable by a processor for implementing specific logical functions orsteps in the process. The program code may be stored on any type ofcomputer readable medium, for example, such as a storage deviceincluding a disk or hard drive. The computer readable medium may includea non-transitory computer readable medium, for example, such ascomputer-readable media that stores data for short periods of time likeregister memory, processor cache and Random Access Memory (RAM). Thecomputer readable medium may also include non-transitory media, such assecondary or persistent long term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer readable media may also be any other volatile ornon-volatile storage systems. The computer readable medium may beconsidered a computer readable storage medium, a tangible storagedevice, a computer program product, or other article of manufacture, forexample.

The non-transitory computer readable medium could also be distributedamong multiple data storage elements, which could be remotely locatedfrom each other. A computing device that executes some or all of thestored instructions could be a vehicle, such as the example vehicle 200illustrated in FIG. 2. Alternatively, the computing device that executessome or all of the stored instructions could be another computingdevice, such as a server.

In addition, for the method 300 and other processes and methodsdisclosed herein, each block in FIG. 3 may represent circuitry that iswired to perform the specific logical functions in the process.

Example methods, such as method 300 of FIG. 3 may be carried out inwhole or in part by the vehicle and its subsystems. Accordingly, examplemethods could be described by way of example herein as being implementedby the vehicle. However, it should be understood that an example methodmay be implemented in whole or in part by other computing devices of thevehicle or separate from the vehicle. For example, an example method maybe implemented in whole or in part by a server system, which receivesdata from a device such as those associated with the vehicle. Otherexamples of computing devices or combinations of computing devices thatcan implement an example method are possible.

At block 302, the method 300 includes receiving laser data collectedfrom scans of an environment of a vehicle. The laser data includes aplurality of laser data points that are based on objects in theenvironment that are perceived to be physically present due to reflectedor backscattered light. The vehicle, or components of the vehicle suchas a computing device or processor, may be configured to receive laserdata that is collected.

As an example, the vehicle may have a LIDAR unit that illuminates areasaround, surrounding, in front of, behind, to the side, or in anyproximity or relation to the vehicle, and detects reflected light. Inoperation, the LIDAR rotates and (e.g., periodically) emits laser beams.Reflections from the emitted laser beams by objects in the environmentare then received by suitable sensors. Time-stamping receipt of thereflected signals allows for associating each reflected signal (if anyis received at all) with the most recently emitted laser pulse, andmeasuring the time delay between emission of the laser pulse andreception of the reflected light. The time delay provides an estimate ofthe distance to the reflective feature by scaling according to the speedof light in the intervening atmosphere. Combining the distanceinformation for each reflected signal with the orientation of the LIDARdevice for the respective pulse emission allows for determining aposition of the reflective feature in three-dimensions. For illustrativepurposes, an environmental scene can be described in the two-dimensionalx-y plane in connection with a single sweep of the LIDAR device thatestimates positions to a series of points located in the x-y plane.However, it is noted that a more complete three-dimensional sampling isprovided by either adjusting beam steering optics to direct the laserbeam up or down from the x-y plane on its next sweep of the scene or byproviding additional lasers and associated beam steering opticsdedicated to sampling point locations in planes above and below the x-yplane, or combinations of these.

At block 304, the method 300 includes associating, by a computingdevice, laser data points of the plurality of laser data points with oneor more objects in the environment. As an example, a tracking system maybe used to track objects, and laser data points that are associated withtracked objects can be determined. Laser data received due to thereflective features in an environment may be deemed to be due to anobject physically present in the environment (e.g., causing the laser tobe reflected). The computing device can be configured to storeassociated locations of such objects seen by the laser data, and tostore associations between the laser data and the objects.

In some examples, a point cloud corresponding to objects in theenvironmental can be generated. Each point in the point cloud can bereferenced by an azimuth angle (e.g., orientation of the LIDAR devicewhile emitting the pulse corresponding to the point, which is determinedby the orientation of an rotating angled mirror of the LIDAR) and aline-of-sight (LOS) distance (e.g., a distance indicated by the timedelay between pulse emission and reflected light reception). For pulsesthat do not result in a returning reflected signal, the distance in thepoint map can optionally be set to the maximum distance sensitivity ofthe LIDAR device. The maximum distance sensitivity can be determinedaccording to the maximum time delay the associated optical sensors waitfor a return reflected signal following each pulse emission, which canitself be set according to the anticipated signal strength of areflected signal at a particular distance given ambient lightingconditions, intensity of the emitted pulse, predicted reflectivity ofenvironmental features, etc. In some examples, the maximum distance canbe approximately 60 meters, 80 meters, 100 meters, or 150 meters, butother examples are possible for particular configurations of the LIDARdevice and associated optical sensors.

In some embodiments, the sensor fusion algorithm 138, computer visionsystem 140, and/or computer system 112 illustrated in FIG. 1, can beconfigured to interpret the collected laser data alone and/or incombination with additional sensor-indicated information and/ormemory-based pattern-matching point clouds and/or baseline maps of theenvironment to categorize or identify group of points as correspondingto objects in the environment.

Further, each spatial point can be associated with a respective laserfrom a set of lasers and a respective timestamp. That is, in anembodiment where the LIDAR includes multiple lasers, each respectivereceived spatial point can be associated with the particular laser thatwas detected in accordance with the respective received spatial point.Additionally, each respective spatial point can be associated with arespective timestamp (e.g., a time at which laser was emitted orreceived). In this way, the received spatial points may be organized,identified, or otherwise ordered on a spatial (laser identification)and/or temporal (timestamp) basis. Such an ordering may assist orimprove an analysis of the spatial-point data by allowing for organizingthe spatial-point data into a meaningful order.

In some examples, object detection is provided in connection with anexample LIDAR device. The LIDAR device may be configured to capturelaser point cloud images using one or more lasers. The laser point cloudincludes many points for each pulse emitted from the LIDAR device;reflected signals may indicate actual locations of reflective objects,whereas failing to receive reflected signals indicate an absence ofsufficiently reflective objects within a particular distance along theline of sight of the laser. Depending on factors including the laserpulse rate, the scene refresh rate, the total solid angle sampled byeach LIDAR device (or just the total solid angle of the scene, whereonly one LIDAR device is used), the number of sample points in eachpoint cloud can be determined. Some embodiments can provide point cloudswith as many as 50,000 laser-indicated points, 80,000 laser-indicatedpoints, 100,000 laser-indicated points, etc. Generally, the number oflaser-indicated points in each point cloud is a tradeoff between angularresolution on the one hand, and refresh rate on the other hand. TheLIDAR device is driven to provide an angular resolution at asufficiently high refresh rate to be relevant to real time navigationaldecisions for an autonomous vehicle. Thus, the LIDAR device can beconfigured to capture one or more laser point clouds of the scanningzone at predetermined time intervals, such as 100 milliseconds (toachieve a refresh rate of 10 frames per second), 33 milliseconds (toachieve a refresh rate of 30 frames per second), 1 millisecond, 1second, etc., so as to capture a number of scans of the environment.

With reference to FIG. 1, data storage 114 of computer system 112 ofvehicle 100 can store object-detector software, code, or other programinstructions. Such object-detector software can include, or be part of,one or more of the control systems 106 described above, including thesensor fusion algorithm 138, computer vision system 140, and/or obstacleavoidance system 144. The object detector may be any configuration ofsoftware and/or hardware configured to perceive features in theenvironmental scene by categorizing and/or identifying objects based onthe laser point clouds captured by the LIDAR 128 and/or based on one ormore of the sensors in sensor system 104. As a laser point cloud iscaptured via LIDAR 128, data indicative of the captured point cloud iscommunicated to the object detector, which analyzes the data todetermine whether there is an object present in the laser point cloud.Objects indicated by the point cloud may be, for example, a vehicle, apedestrian, a road sign, a traffic light, a traffic cone, etc.

To determine whether an object is present in a laser point cloud image,the object detector software and/or module can associate arrangements oflaser-indicated points with patterns matching objects, environmentalfeatures, and/or categories of objects or features. The object detectorcan be pre-loaded (or dynamically instructed) to associate arrangementsaccording to one or more parameters corresponding to physicalobjects/features in the environment surrounding the vehicle 100. Forexample, the object detector can be pre-loaded with informationindicating a typical height of a pedestrian, a length of a typicalautomobile, confidence thresholds for classifying suspected objects,etc.

When the object detector identifies an object in point cloud, the objectdetector can define a bounding box encompassing that object. Forexample, the bounding box can correspond to a predicted exterior surfaceof the point cloud indicated object. Of course, the bounding “box” cangenerally take the form of a multi-sided closed shape defining thepredicted outer boundaries of the object.

For each captured point cloud, positions of perceived objects and theircorresponding boundary definitions are associated with a frame number orframe time. Thus, similarly shaped objects appearing in roughly similarlocations in successive scans of the scene can be associated with oneanother to track objects in time. For perceived objects appearing inmultiple point cloud frames (e.g., complete scans of the scanning zone),the object can be associated, for each frame on which the objectappears, with a distinct bounding shape defining the dimensional extentof the perceived object.

Perceived objects can be tracked as the vehicle 100 travels through itssurrounding environment and/or as objects move with respect to thevehicle so as to pass through the scanning zone of the LIDAR device 128.Combining two or more successively captured point clouds can therebyallow for determining translation information for detected objects.Future position predictions can be made for objects with characterizedmotion profiles, such as by observing acceleration and/or velocity ofobjects such as cars moving along the roadway with the vehicle 100 topredict the location of the object during a subsequent scan. In someembodiments, objects moving through the air are assumed to move along atrajectory influenced by the force of gravity.

To assist in providing object recognition, the vehicle 100 can also bein communication with an object-identification server (e.g., via thewireless communication system 146). The object-identification server canverify and/or classify objects detected by vehicle 100 using the objectdetector. Moreover, the object-identification server can facilitateoptimization of one or more of the parameters used by the objectdetector to detect objects in the captured laser point cloud based onaccumulated data from other similar systems, local conditions. In oneembodiment, vehicle 100 can communicate the object boundaries, and theircorresponding object parameters, to the object identification server forverification that the perceived objects are correctly identified, suchas indicated by an evaluation for statistical likelihood of correctidentification.

Referring back to FIG. 3, at block 306, the method 300 includescomparing laser data points that are unassociated with the one or moreobjects in the environment with stored laser data points representativeof a pattern due to fog. Within examples, for laser data that is deemedto be representative of an object, the laser data may be associated withthe object in the environment, and remaining received or collected laserdata may be considered unassociated with the objects in the environment.

Within examples, the method 300 may be executed to determine instancesin which objects are not detected by the laser that are expected to bepresent or objects are detected that are unexpected (i.e., lasermeasurements are lost or altered due to the fog). As described above,for received reflected laser data, it may be determined that an objectis present to cause the laser to be reflected. But, in foggy conditions,the fog may cause a reflection, resulting in a false determination of anobject. To determine when laser data is based on real objects present inan environment versus a fog that causes a reflection of laser emissions,the computing device can be configured to compare laser data with laserdata known to be representative of objects and/or known to be due to fogreflections. Thus, comparison of received laser data with patterns oflaser data known to be representative of objects can be made to identifylaser data likely to be due to something other than an object.

Any number of patterns of laser data may be stored and referenced forcomparison to the unassociated laser data to categorize the unassociatedlaser data as being representative of an object or due to a weathercondition, such as fog.

As another example to distinguish between laser data due to objects anddue to other conditions (e.g., a fog), the computing device may beconfigured to track movement of objects as represented by laser data.Thus, all laser data may be tracked as being representative of movingobjects, and objects that are seen to move in an odd manner, or thatappear and disappear over a short time frame, may be considered falseobjects that are due to laser data reflections caused by a weathercondition.

Within examples, the method 300 may include determining the laser datapoints that are unassociated with the one or more objects in theenvironment. The LIDAR may be configured to perform a first scan of theenvironment, and associating the laser data points of the plurality oflaser data points with one or more objects in the environment. The LIDARmay then be configured to perform a second scan of the environment, anddetermine laser data points that match to the one or more objects basedon a location of an object represented by the laser data points. Then,laser data points of the second scan that are unassociated with the oneor more objects in the environment based on a lack of a match to the oneor more objects in the first scan can be determined. In furtherexamples, the second scan may be representative of a scan of theenvironment after the first scan or after a number of scans so thatobject tracking has occurred over some time for comparison to the secondscan data.

At block 308, the method 300 includes based on the comparison,identifying by the computing device an indication that a weathercondition of the environment of the vehicle includes fog. In an example,when the unassociated laser data matches or substantially matches to apattern of laser data due to fog, the computing device may identify thatthe laser data is due to fog.

In one example, laser data may relate to presence of water in the air orcloudy conditions typical with a fog, and such water droplets or cloudsare untracked items by the vehicle's tracking system. For example, to aLIDAR device, which detects and locates optically reflective features,water droplets in the air or cloud features in the air resemble anobject cloud because light pulses are reflected from the particulates.The LIDAR device may not be able to penetrate through a fog, and thus,when a fog is present, many laser points may be returned due to the fog.Thus, for any untracked received laser data (e.g., laser data that doesnot match to tracked objects), an indication can be determined that aweather condition of the environment of the vehicle includes fog.

In addition (or alternatively), at block 306, the method 300 may includedetermining that a number of laser data points that are unassociatedwith the one or more objects in the environment exceeds a predefinedthreshold, and then identifying by the computing device an indicationthat a weather condition of the environment of the vehicle includes fog.For instance, if there are a few laser data points unassociated withobjects, this may be due to any number of factors (such as spuriousreflected lasers for instance); however, for a certain threshold numberof laser data points, a higher probability may exist that these are dueto emitted laser beams reflected off of water droplets in the air.

In another example, as a vehicle's laser or lasers are moved along, thevehicle may collect data points including range and intensityinformation for the same location (point or area) from severaldirections and/or at different times. For example, each data point mayinclude an intensity value indicative of reflectivity of an object fromwhich light was received by the laser as well as location information.Highly reflective surfaces may be associated with an intensity valuewhich is greater than less reflective surfaces. Similarly, darkerobjects (black, navy blue, brown, etc.) which absorb more light may beassociated with a lower intensity value than lighter colored objectswhich may reflect more light (white, cream, silver, etc.). In thisregard, cloudy fogs may present a darker environment, and therefore,rather than increasing the intensity value of a perceived object, a fogmay decrease the intensity value.

In some examples, intensity values of laser data point may be scaled,for example, from 0-250, where 0 is dark and 250 is bright. Thus, morereflective, brighter surfaces may be associated with intensity valuescloser to 250, while less reflective, darker surfaces may be associatedwith intensity values closer to 0. A laser scan data may be received andprocessed to generate geographic location coordinates. These geographiclocation coordinates may include GPS latitude and longitude coordinates(x,y) with an elevation component (z), or may be associated with othercoordinate systems. A result of this processing is a set of data point.Each data point of this set may include an intensity value indicative ofreflectivity of the object from which the light was received by thelaser as well as location information: (x,y,z).

An average intensity of the laser data points for an environment may becompared to a threshold to further identify an indication of a fog. Forexample, as noted above, the water droplets may decrease the intensityof laser data collected. Thus, a foggy environment may have laser datawith average intensity values that are somewhat lower that of a clearenvironment. If some percentage of the examined laser data points haveintensity values below a certain threshold value, a determination thatthe weather condition includes a fog can be made. For example, using the0 to 250 scale described above, if there are 1000 laser data points inthe roadway and at least 850 of these 1000 points (or at least 85% ofthese laser data points) have an intensity below a threshold of 10, theenvironment is dark. As noted above, a lower intensity value mayindicate a high probability that a fog is present. Other thresholds andpercentages may also be used, for example. Laser data representative ofa clear environment may be stored and accessed for comparison purposes,for example.

In further examples, the method 300 may include determining a locationof the vehicle, determining a distribution of expected values of laserdata for the environment of the location of the vehicle, and comparingthe determined distribution to the laser data collected for theenvironment of the vehicle. The distribution of expected values of laserdata for the location for clear conditions may be retrieved from datastorage or received from a server, for example, based on the location ofthe vehicle. Maps of intensity of laser data per location can begenerated and stored for both dry and clear conditions. Based on adifference between the determined distribution and the laser datacollected for the environment of the vehicle being above a threshold, anidentification of a second indication that the weather conditionincludes a fog can be made. For instance, if the difference is high,then the laser data does not match that as expected to be seen for aclear day. Thus, such comparisons can be indicative of anotherindication that a fog is present.

In still further examples, the method 300 may further includedetermining a form of a shape of the laser data points that areunassociated with one or more objects in the environment. A fog may havea unique shape, and shapes and radius's of point clouds generated due togeneral fogs may be created and stored, and later compared to receivedlaser data. When the laser data maps to a stored shape of a fog, then afurther indication of the fog can be made.

Within examples, using the method 300, the vehicle may be configured tooperate in an autonomous mode, and the indication that the surface onwhich the vehicle travels is wet can be utilized to determine a drivingdecision for the vehicle. It may be desired to control the vehicledifferently due to various weather conditions or conditions of the road,and thus, when fog is present, the vehicle may be operated according toa “fog” driving technique (e.g., allow more space for braking, reducespeeds, etc.). Vehicles may not operate as well in certain weatherconditions, and thus, in some examples, based on the indication of afog, the vehicle may provide instructions to indicate a request totransition to a manual mode, such as by providing an alert to the driverto begin operation of the vehicle by disabling the autonomous mode.

In addition or as an alternative to making a driving decision, the roadweather conditions and/or road surface conditions may trigger an alertto be sent to the driver. The alert may request the driver to takecontrol of the vehicle or may simply provide an alert to the driver ofthe condition. The alert may be an aural signal, a visual signal, ahaptic or tactile and/or any other signal that gets the attention of thedriver. In this example, after alerting the driver, the vehicle mayreceive input from the driver, such as turning the steering wheel,applying the brake, applying the accelerator, pressing an emergencyshut-off, etc.

In some examples, additional data may be considered (in addition tolaser data points being unassociated with tracked objects) to make adetermination that a weather condition of the environment in which thevehicle resides includes fog, or to provide a higher probability orconfidence of fog being present (e.g., that the unassociated laser datapoints are due to water particulates in the air versus a trackedobject).

FIG. 4 is a block diagram of example methods for determining furtherindications of weather conditions including fog using onboard vehiclesensors, in accordance with at least some embodiments described herein.Method 400 shown in FIG. 4 presents an embodiment of a method that, forexample, could be used with the vehicle 100 and/or vehicle 200 asillustrated and described in reference to FIGS. 1 and 2, respectively(or components of the vehicle 100 or vehicle 200), and may be executedin addition to the method 300 shown in FIG. 3. The method 400 mayrepresent a module, a segment, or a portion of program code, whichincludes one or more instructions (e.g., machine readable code)executable by a processor for implementing specific logical functions orsteps in the process.

As shown at block 402, the method 400 may include determining whetherthe laser data is in agreement with RADAR data. For example, theautonomous vehicle may collect radar data for the environment of thevehicle, and the RADAR data is indicative of a presence of objects inthe environment of the vehicle. A radar will generally not return datafor water particulates or cloudy compositions present in the air,however, the radar data can identify a objects through the clouds. Thelaser data is also used to track objects as well as described above, andwhen the presence of an object as indicated by the RADAR data does notagree with the presence of an object as indicated by the LIDAR data,then the LIDAR data may be deemed to be incorrect. Any additionalobjects indicated by the LIDAR data that are not indicated by the RADARdata can be indicators of water particulates or other returned laserdata due to a fog. In such examples, when objects due to LIDAR data areseen at a certain distance and detected for some threshold period oftime, but no objects due to RADAR are see that match, an indication thatthe roadway is foggy can be made.

In some examples, any objects tracked by the LIDAR data that are notseen in the RADAR data may be categorized as due to a weather condition.As shown in the method 400, based on a lack of agreement of the RADARand LIDAR data, the method 400 includes determining false returns oflaser data at block 404. At block 406, a determination can be made as towhether a number of the false returns is above a threshold, as discussedabove. For example, for a small number of false returns, such asspurious or erroneous reflected laser data, no further action may betaken or the laser data can be reprocessed (as shown at block 414).However, when the number of false returns is above the threshold, themethod 400 includes identifying an indication that a weather conditionof the environment of the vehicle includes fog, as shown at block 408.

In further examples, laser data points that are unassociated withobjects as indicated by the RADAR that form a circular shape aroundobjects indicated by the RADAR can be determined to be due to fog. Insome examples, a density of the fog may also be estimated. For example,a distance between an object in the environment that is unassociatedwith laser data points (e.g., a fog) and a given object in theenvironment indicated by the radar data (and also possibly associatedwith laser data points) can be determined, and an estimation of adensity of the fog can be made based on the distance. Using the twosensors (LIDAR and RADAR), a presence of an object at a certain rangecan be detected, and the distance between where the fog is detected towhere the object is located can provide a measurement of visibility ordensity of the fog. Density may be stated as a distance of visibility,such as visibility of less than 5 km, for example, and the distance canbe that as determined between where the fog is detected and where theobject is located. In still further example, a determination of adensity of the fog may be based on a number of unassociated laser datapoints. In instances in which a heavy or dense fog is present, there maybe a large number of unassociated laser data points.

The method 400 may proceed to block 410 where the method 400 includesdetermining whether the laser data is in agreement with camera data. Forexample, the autonomous vehicle may include a camera and may receiveimage data collected for the environment of the vehicle. A fog may bevisible in images, and thus, images may be processed using objectdetection techniques, e.g., by providing the images to an objectdetection server and receiving a response indicating objects in theimages.

FIG. 5A is an example conceptual illustration of identifying anindication that a weather condition of the environment includes fog. InFIG. 5A, a vehicle 502 travels on a surface and includes sensors thatcollect data of areas around the vehicle 502. The sensors may include aLIDAR unit that can receive reflected laser beams from areas in theenvironment and a camera that can be configured to collect images of theenvironment. Another vehicle 504 may be traveling in front of thevehicle 502. The camera may capture an image of the area 506.

FIG. 5B is an example conceptual illustration of an image captured bythe vehicle 502 in FIG. 5A. The image illustrates that a portion of arear of the vehicle is covered or obscured by a fog. The image may beprocessed to determine that a number of pixels in the image data aresubstantially grey in color. Based on a percentage of the pixels beingsubstantially grey in color, a second indication of a foggy weathercondition may be made. In FIG. 5B, about 40-45% of the image is obscuredby the fog, and thus, the threshold may be met. Lower percentages may beused as well. However, percentages of about 40% or more may be used forhigher probabilities of the image including a fog.

In some examples, the image data may be processed to identify objects inthe image, and when the objects as indicated by the camera image do notagree with objects as indicated by the LIDAR data, an indication that afog is present can be made as well. For example, the image in FIG. 5Bmay indicate an object present in an area covering coordinates x₁-x_(n),while the laser data may only indicate an object present in an areacovering coordinates x₂-x_(n). Thus, the camera data does not agree withthe laser data, and such disagreement can be used as an indicator of afog being present.

Referring back to FIG. 4, based on a disagreement of the camera data andthe laser data, an indication of the environment of the vehicleincluding fog can be made, as shown at block 408.

The method 400 may proceed to block 412 where the method 400 includesdetermining whether the laser data is in agreement with weather data.For example, weather information for a location of the vehicle can bereceived from a server over a network, and when the weather informationindicates a chance of fog, or foggy conditions, a second indication thata fog is present can be made. Weather data may alternatively oradditionally be received from onboard vehicle sensors, such as aprecipitation sensor or “rain detector”, and when the LIDAR data isindicative of false returns combined with the precipitation sensorindicating the presence of precipitation, a second indication can bemade that the fog is present. It may not be possible to rely solely onoutputs of the precipitation sensor to determine whether the atmosphereincludes a sufficient amount of moisture to cause a fog, and thus, thesensor outputs can be used as a secondary measure.

In further examples, additional weather data such as a temperature ordew point, can be determined from on-board vehicle sensors orcommunication with a server, and a second indication that the weathercondition of the environment of the vehicle includes the fog can be madebased on the temperature or dew point. For example, a fog may form whena difference between temperature and dew point is about less than 2.5degrees Celsius or 4 degrees Fahrenheit. Fogs normally occur at arelative humidity near 100%, however, fogs can form at lower humidities,and fog can sometimes not form with relative humidity at 100%. A readingof 100% relative humidity means that the air can hold no additionalmoisture; the air will become supersaturated if additional moisture isadded. Fogs can form suddenly, and can dissipate just as rapidly,depending what side of the dew point the temperature is on.

Within examples, the method 400 may be performed to provide furtherindications that the environment of the vehicle includes fog. The method400 may be optionally performed in addition to the method 300 in FIG. 3.For example, the unassociated laser points (e.g., generated due to waterdroplets, moisture in the air, or generally fog) could be unassociatedrelative to laser-detected vehicles or possibly unassociated relative toradar-detected vehicles, and the basis for identifying that the fog maybe verified by cross-reference to other sensor data, weatherinformation, etc. Any combination of the functions described in FIG. 4may be combined with FIG. 3 to provide an improved confidence level thata fog is present.

Still further, in addition to the sensor data described above, inputfrom other sensors on vehicles can be used to make additionaldeterminations of weather conditions. For example, these sensors mayinclude tire pressure sensors, engine temperature sensors, brake heatsensors, brake pad status sensors, tire tread sensors, fuel sensors, oillevel and quality sensors, air quality sensors (for detectingtemperature, humidity, or particulates in the air), etc.

Further, many of the sensors on the vehicle provide data that isprocessed in real-time, that is, and the sensors may continuously updateoutputs to reflect the environment being sensed at or over a range oftime, and continuously or as-demanded.

FIGS. 6A-6B include example conceptual side view illustrations ofidentifying an indication that an environment of a vehicle includes afog. In FIG. 6A, a vehicle 602 travels on a surface and another vehicle604 travels in front of the vehicle 602. The vehicle 602 may include aLIDAR unit 606 that is configured to receive laser data collected for anarea in front of the vehicle 602 (e.g., as shown by the arrows in FIG.6A), and the laser data may indicate that the vehicle 604 is a atdistance d₁ from the vehicle 602. The vehicle 604 may travel through afog, as shown in FIG. 6B. The water in the fog may be detected by theLIDAR unit 606 when laser beams are reflected by the water, and thelaser data may now indicate that the vehicle 604 is at a distance d₂from the vehicle 602 due to the fog. However, the water may not be anobject tracked by a tracking system of the vehicle 602, since othersensors (e.g., RADAR) may not detect the water or the LIDAR unit 608 maynot track the water constantly over time. When laser data is receivedthat does not match to tracked objects, the unassociated laser data canbe deemed an indication of water in the air, and an indication that afog is present.

The laser data may be compared to a predefined or stored shape pointcloud that may be generated due to a fog. As an example, laser datapoints that are approximately around a tracked object, as indicated bythe radar data, that are representative of an objects of a substantiallycircular shape that has a radius of a fog (when compared to stored pointclouds) can be processed by comparison to stored point clouds forfurther verifications of the fog. For all laser data points unassociatedwith an object in the environment indicated by the radar data that arerepresentative of objects approximately around a tracked object, suchlaser data points may be indicative of the fog. In this regard, thevehicle 602 may detect a cloud or clouds of random data points at a rearof the vehicle 604, and as a consequence of the moving water in the air,a location and number of the data points within the clouds mayconstantly be changing. Thus, a cloud of data points from a fog may nothave a definitive structure, whereas a portion of solid object, such asthe rear end of a vehicle, would be associated with data points defininga clear surface. Similar data point clouds may also be observed behindother vehicles. Such observations may indicate that a fog is present.

It should be understood that arrangements described herein are forpurposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g. machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location, or other structural elementsdescribed as independent structures may be combined.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

What is claimed is:
 1. A method comprising: receiving laser datacollected from scans of an environment of a vehicle, wherein the laserdata includes a plurality of laser data points; determining, by acomputing device, laser data points of the plurality of laser datapoints that are associated with one or more tracked objects in theenvironment; determining, by the computing device, laser data points ofthe plurality of laser data points that are unassociated with the one ormore tracked objects in the environment; comparing the laser data pointsthat are unassociated with the one or more tracked objects in theenvironment with stored laser data points representative of a patterndue to fog; and based on the comparison, identifying by the computingdevice an indication that a weather condition of the environment of thevehicle includes fog.
 2. The method of claim 1, further comprising:tracking one or more objects in the environment of the vehicle based onoutputs of sensors of the vehicle.
 3. The method of claim 1, furthercomprising: for a first scan of the environment, associating, by thecomputing device, the laser data points of the plurality of laser datapoints with one or more objects in the environment; for a second scan ofthe environment, determining laser data points that match to the one ormore objects based on a location of an object represented by the laserdata points; and determining the laser data points that are unassociatedwith the one or more tracked objects in the environment based on a lackof a match to the one or more objects in the first scan.
 4. The methodof claim 1, wherein identifying by the computing device the indicationthat the weather condition of the environment of the vehicle includesfog comprises: determining that a number of the laser data pointsunassociated with the one or more tracked objects in the environment isabove a threshold number.
 5. The method of claim 1, further comprising:receiving additional data collected for the environment of the vehiclefrom one or more additional sensors, wherein the additional data isindicative of a presence of one or more objects in the environment ofthe vehicle; determining laser data points of the plurality of laserdata points that are associated with the one or more objects in theenvironment indicated by the additional data; and based on laser datapoints being unassociated with the one or more objects in theenvironment indicated by the additional data, identifying the indicationthat the weather condition of the environment of the vehicle includesfog.
 6. The method of claim 1, further comprising: receiving image data;and identifying a second indication that the weather condition of theenvironment of the vehicle includes fog based on the image data.
 7. Themethod of claim 6, wherein identifying the second indication that theweather condition of the environment of the vehicle includes fog basedon the image data comprises: determining that a number of pixels in theimage data are substantially grey in color.
 8. The method of claim 1,further comprising: receiving image data collected for the environmentof the vehicle; determining, by the computing device, one or moreobjects in the image data that are associated with the one or moretracked objects in the environment; and based on an object of the one ormore objects in the image data being unassociated with the one or moretracked objects in the environment, identifying by the computing devicea second indication that the weather condition of the environment of thevehicle includes fog.
 9. The method of claim 1, wherein the vehicle isconfigured to operate in an autonomous mode, and the method furthercomprises based on the indication that the weather condition of theenvironment of the vehicle includes fog, determining a driving decisionfor the vehicle.
 10. The method of claim 1, wherein the vehicle isconfigured to operate in an autonomous mode, and the method furthercomprises based the indication that the weather condition of theenvironment of the vehicle includes fog, providing instructions toindicate a request to transition to a manual mode.
 11. The method ofclaim 1, further comprising: receiving weather information for alocation of the vehicle from a server over a network; and identifying asecond indication that the weather condition of the environment of thevehicle includes the fog based on the weather information.
 12. Themethod of claim 1, further comprising: receiving a current temperaturefor a location of the vehicle; and identifying a second indication thatthe weather condition of the environment of the vehicle includes fogbased on the current temperature.
 13. A method comprising: receivinglaser data collected from scans of an environment of a vehicle, whereinthe laser data includes a plurality of laser data points; receivingradar data collected for the environment of the vehicle, wherein theradar data is indicative of a presence of one or more objects in theenvironment of the vehicle; determining, by a computing device, laserdata points of the plurality of laser data points that are associatedwith the one or more objects in the environment indicated by the radardata; and based on laser data points being unassociated with the one ormore objects in the environment indicated by the radar data, identifyingby the computing device an indication that a weather condition of theenvironment of the vehicle includes fog.
 14. The method of claim 13,further comprising: associating, by the computing device, laser datapoints of the plurality of laser data points with one or more objects inthe environment; comparing laser data points that are unassociated withthe one or more objects in the environment with stored laser data pointsrepresentative of a pattern due to fog; and based on the comparison,identifying by the computing device the indication that the weathercondition of the environment of the vehicle includes fog.
 15. The methodof claim 13, further comprising: determining an object represented bylaser data points that are unassociated with the one or more objects inthe environment indicated by the radar data; determining a distancebetween the object and a given object of the one or more objects in theenvironment indicated by the radar data and associated with laser datapoints; and determining an estimation of a density of the fog based onthe distance.
 16. The method of claim 13, wherein identifying by thecomputing device the indication that the weather condition of theenvironment of the vehicle includes fog comprises: determining that thelaser data points unassociated with the one or more objects in theenvironment indicated by the radar data are representative of datapoints approximately around one of the one or more objects indicated bythe radar data.
 17. The method of claim 16, further comprisingdetermining that the data points approximately around one of the one ormore objects indicated by the radar data are representative of asubstantially circular shape having a radius of a fog.
 18. The method ofclaim 13, wherein the vehicle is configured to operate in an autonomousmode, and the method further comprises based on the indication that theweather condition of the environment of the vehicle includes fog,transitioning to a manual mode.
 19. A system, comprising: at least oneprocessor; and non-transitory data storage comprising instructionsexecutable by the at least one processor to cause the system to performfunctions comprising: receiving laser data collected from scans of anenvironment of a vehicle, wherein the laser data includes a plurality oflaser data points; associating laser data points of the plurality oflaser data points with one or more objects in the environment; comparinglaser data points that are unassociated with the one or more objects inthe environment with stored laser data points representative of apattern due to fog; based on the comparison, identifying by thecomputing device an indication that a weather condition of theenvironment of the vehicle includes fog; receiving radar data collectedfor the environment of the vehicle, wherein the radar data is indicativeof a presence of one or more objects in the environment of the vehicle;determining an object in the environment indicated by the radar datathat is unassociated with the laser data points; determining a distancebetween the object in the environment indicated by the radar dataunassociated with laser data points and a given object of the one ormore objects in the environment indicated by the radar data andassociated with laser data points; and determining an estimation of adensity of the fog based on the distance.
 20. The system of claim 19,wherein the vehicle is configured to operate in an autonomous mode, andthe functions further comprise: based on the indication that the weathercondition of the environment of the vehicle includes fog, determining adriving decision for the vehicle; and providing instructions to indicatea request to transition to a manual mode.